Papers with reading comprehension
Answering Naturally: Factoid to Full length Answer Generation (D19-54)
Copied to clipboard
| Challenge: | Factoid question answering systems extract answers for a question from passages, which are usually short spans of text . but, these spans would result in an unnatural reading experience in a conversational system . a pointer generator based full-length answer generator can be used with most QA systems . |
| Approach: | They propose a pointer generator based full-length answer generator which can be used with most QA systems. |
| Outcome: | The proposed system generates full length answer without relying on passage from which it was extracted. |
Is It Dish Washer Safe? Automatically Answering “Yes/No” Questions Using Customer Reviews (N19-3)
Copied to clipboard
| Challenge: | Using Amazon reviews, we find that the answer to a question is only in 45% of cases. |
| Approach: | They combine Amazon reviews with consumer reviews and manually analyse 400 questions from four domains to find that reviews directly contain the answer to the question . they then compare QA systems that use reviews in addition to the questions to see if they can be useful for other question types. |
| Outcome: | The proposed system outperforms the chance baseline but not by a large margin. |
AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models (D19-3)
Copied to clipboard
| Challenge: | Existing interpretation codebases make it difficult to apply these methods to new models and tasks. |
| Approach: | They propose a framework for interpreting NLP models that provides explanations for specific models. |
| Outcome: | The proposed framework provides interpretation primitives for any AllenNLP model and task, a suite of built-in interpretation methods, and a library of front-end visualization components. |
TextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processing (2020.acl-demos)
Copied to clipboard
| Challenge: | Large pre-trained language models have hundreds of millions of parameters and take several gigabytes of memory to train and inference. |
| Approach: | They propose an open-source knowledge distillation toolkit designed for natural language processing that provides a set of predefined distillation methods and can be extended with custom code. |
| Outcome: | The proposed method is comparable with or even higher than the public distilled BERT models with similar numbers of parameters. |
Analyzing Wrap-Up Effects through an Information-Theoretic Lens (2022.acl-short)
Copied to clipboard
| Challenge: | a lack of studies targeting naturalistic sentence-final reading behavior is likely to explain the lack of data on reading time (RT) data is omitted due to the confounding factors introduced by so-called "wrap-up effects" |
| Approach: | They propose to look for a link between “wrap-up effects” and information theoretic quantities such as word and context information content. |
| Outcome: | The proposed model omits data on words at the end of sentences or clauses to control for the confounding factors introduced by wrap-up effects. |
PrimeQA: The Prime Repository for State-of-the-Art Multilingual Question Answering Research and Development (2023.acl-demo)
Copied to clipboard
Avi Sil, Jaydeep Sen, Bhavani Iyer, Martin Franz, Kshitij Fadnis, Mihaela Bornea, Sara Rosenthal, Scott McCarley, Rong Zhang, Vishwajeet Kumar, Yulong Li, Md Arafat Sultan, Riyaz Bhat, Juergen Bross, Radu Florian, Salim Roukos
| Challenge: | Question Answering (QA) is a major area of research in Natural Language Processing (NLP) |
| Approach: | They propose a one-stop and open-source QA repository for question answering . it supports core QA functionalities like retrieval and reading comprehension . they say it will facilitate easy replication of state-of-the-art (SOTA) QA methods . |
| Outcome: | The proposed framework enables easy replication of state-of-the-art (SOTA) QA methods. |
AnswerQuest: A System for Generating Question-Answer Items from Multi-Paragraph Documents (2021.eacl-demos)
Copied to clipboard
| Challenge: | Existing systems that generate and answer questions in a question-and-answer format can facilitate reading comprehension. |
| Approach: | They propose a system that integrates question answering and question generation tasks to produce a list of Q&A items for a text. |
| Outcome: | The proposed system generates a catalog of Q&A items for a text. |
Identifying Supporting Facts for Multi-hop Question Answering with Document Graph Networks (D19-53)
Copied to clipboard
| Challenge: | Recent advances in reading comprehension have resulted in models that surpass human performance when the answer is contained in a single, continuous passage of text. |
| Approach: | They propose a document-structured message passing architecture for the identification of supporting facts over a graph-structure based representation of text. |
| Outcome: | The proposed model outperforms a baseline reading comprehension test on raw text and shows that it is relevant for multi-hop reasoning. |
Read and Comprehend by Gated-Attention Reader with More Belief (N18-4)
Copied to clipboard
| Challenge: | Existing approaches to read comprehension using gated-attention have been effective . collaborative gating and self-belief aggregation are proposed to address these assumptions . |
| Approach: | They propose to use a document-to-query attention system to gate token encodings of a query . they conjecture that query tokens other than the cloze token may be informative . |
| Outcome: | The proposed approaches advance the state-of-the-art results in CNN, Daily Mail, and Who Did What public test sets. |
Learning to Search in Long Documents Using Document Structure (C18-1)
Copied to clipboard
| Challenge: | Reading comprehension models are dominated by recurrent neural networks (RNNs) as documents become longer and questions become complex, sequential reading becomes a significant bottleneck. |
| Approach: | They propose a reading comprehension framework that uses document trees to model an agent that interleaves quick navigation with more expensive answer extraction. |
| Outcome: | The proposed model improves question answering performance compared to existing models and has a strong information-retrieval baseline. |
Active Reading Comprehension: A Dataset for Learning the Question-Answer Relationship Strategy (P19-2)
Copied to clipboard
| Challenge: | Literature in quality learning suggests that task performance should also be evaluated on the undergone process to answer. |
| Approach: | They propose to use the Question-Answer Relationship (QAR) to evaluate a reader's ability to select different sources of information depending on the question type. |
| Outcome: | The proposed model will be used to evaluate reading comprehension with weak supervision. |
On Making Reading Comprehension More Comprehensive (D19-58)
Copied to clipboard
| Challenge: | Getting machines to "understand" text is a vast and long-standing problem, made more challenging by the fact that it is not even clear what it means to understand text. |
| Approach: | They propose a question-based approach to machine reading comprehension that uses a natural language question to test a system's comprehension of a passage of text. |
| Outcome: | The proposed questions have surface cues or other biases that allow a model to shortcut the intended reasoning process. |
Towards Multi-Modal Text-Image Retrieval to improve Human Reading (2021.naacl-srw)
Copied to clipboard
| Challenge: | In primary school, children's books, as well as in modern language learning apps, multi-modal learning strategies like illustrations of terms and phrases are used to support reading comprehension. |
| Approach: | They propose to use multi-modal transformers to train multi-dimensional models on text-image retrieval to support a user's reading comprehension of arbitrary text. |
| Outcome: | The proposed model performs poorly because of the short and relatively simple textual data that the current models are trained with. |
Improving the Robustness of QA Models to Challenge Sets with Variational Question-Answer Pair Generation (2021.acl-srw)
Copied to clipboard
| Challenge: | Existing data augmentation methods for reading comprehension lack robustness to challenge sets whose distribution is different from that of training sets. |
| Approach: | They propose a question-answer pair generation method that generates multiple diverse QA pairs from a paragraph to mitigate this problem. |
| Outcome: | The proposed model improves the accuracy of 12 challenge sets and the in-distribution accuracy. |
Generalizing Question Answering System with Pre-trained Language Model Fine-tuning (D19-58)
Copied to clipboard
| Challenge: | Existing methods focus on improving in-domain performance, leaving open the question of how they can generalize to out-of-domain and unseen RC tasks. |
| Approach: | They propose a multi-task learning framework that learns the shared representation across different tasks and builds on a large pre-trained language model and fine-tuned on multiple RC datasets. |
| Outcome: | The proposed framework improves the BERT-Large baseline by 8.39 and 7.22 respectively. |
Learning to Attend On Essential Terms: An Enhanced Retriever-Reader Model for Open-domain Question Answering (N19-1)
Copied to clipboard
| Challenge: | Existing approaches to open-domain question answering struggle to retrieve indirectly related evidence when no direct evidence is provided. |
| Approach: | They propose a retriever-reader model that learns to attend on essential terms during the question answering process. |
| Outcome: | The proposed model achieves the state-of-the-art on multiple open-domain QA datasets and achieves a 'reader-reader' level. |
EchoPrompt: Instructing the Model to Rephrase Queries for Improved In-context Learning (2024.naacl-short)
Copied to clipboard
| Challenge: | Language models are adopting inference-time prompting techniques such as zero-shot and few-shot prompting. |
| Approach: | They propose a prompting technique that prompts the model to rephrase its queries before answering them. |
| Outcome: | The proposed prompt improves zero-shot-CoT performance of code-davinci-002 by 5% . the proposed prompt is tailored for four scenarios in both zero- and few-shot settings . |
SOCIALITE-LLAMA: An Instruction-Tuned Model for Social Scientific Tasks (2024.eacl-short)
Copied to clipboard
Gourab Dey, Adithya V Ganesan, Yash Kumar Lal, Manal Shah, Shreyashee Sinha, Matthew Matero, Salvatore Giorgi, Vivek Kulkarni, H. Schwartz
| Challenge: | Social science NLP tasks require large data to capture semantics and implicit pragmatics. |
| Approach: | They propose an open-source instruction tuning tool for social science NLP tasks that captures implicit pragmatic cues from text. |
| Outcome: | The proposed model matches or improves on a state-of-the-art, multi-task finetuned model on 80% of social tasks. |
Arithmetic-Based Pretraining Improving Numeracy of Pretrained Language Models (2023.starsem-1)
Copied to clipboard
| Challenge: | Recent work suggests that pretrained language models perform below their capabilities when applied out-of-the-box on tasks that require understanding and working with numbers. |
| Approach: | They propose an extended pretraining approach that addresses both in one extended step . they propose a novel extended pre training objective called Inferable Number Prediction Task to improve numeracy. |
| Outcome: | The proposed approach improves reading comprehension and inference-on-tables tasks without architectural changes or pretraining from scratch. |
GlotEval: A Test Suite for Massively Multilingual Evaluation of Large Language Models (2025.emnlp-demos)
Copied to clipboard
Hengyu Luo, Zihao Li, Joseph Attieh, Sawal Devkota, Ona de Gibert, Xu Huang, Shaoxiong Ji, Peiqin Lin, Bhavani Sai Praneeth Varma Mantina, Ananda Sreenidhi, Raúl Vázquez, Mengjie Wang, Samea Yusofi, Fei Yuan, Jörg Tiedemann
| Challenge: | Existing evaluation frameworks focus on English and a handful of high-resource languages, thereby overlooking the realistic performance of large language models in multilingual and lower-resourced scenarios. |
| Approach: | They propose a unified and lightweight framework that integrates 27 benchmarks under a standard ISO 639-3 language identifier system to enable seamless incorporation of new benchmarks. |
| Outcome: | The proposed framework integrates 27 benchmarks under a standard ISO 639-3 language identifier system, allowing for seamless incorporation of new benchmarks. |
Open-Domain Question Answering Goes Conversational via Question Rewriting (2021.naacl-main)
Copied to clipboard
Raviteja Anantha, Svitlana Vakulenko, Zhucheng Tu, Shayne Longpre, Stephen Pulman, Srinivas Chappidi
| Challenge: | Existing large-scale benchmarks for conversational QA limit the topic of conversation to the content of a single document. |
| Approach: | They propose a dataset for Question Rewriting in Conversational Context (QReCC) the dataset contains 14K conversations with 80K question-answer pairs. |
| Outcome: | The proposed approach shows that the first baseline for the QReCC dataset is 19.10, compared to the human upper bound of 75.45, indicating the difficulty of the setup and a large room for improvement. |
Unsupervised Distractor Generation via Large Language Model Distilling and Counterfactual Contrastive Decoding (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies show that large-scale models can generate unsupervised DG without expensive distractor annotations. |
| Approach: | They propose a dual task training framework that integrates pseudo distractors from LLMs and answer information as the objective target with a two-stage training process. |
| Outcome: | The proposed method surpasses GPT-3.5-turbo zero-shot performance with 200 fewer model parameters. |
The Web as a Knowledge-Base for Answering Complex Questions (N18-1)
Copied to clipboard
| Challenge: | Recent work on reading comprehension made headway in answering simple questions, but tackling complex questions is still an ongoing research challenge. |
| Approach: | They propose to decompose complex questions into a sequence of simple questions and compute the final answer from the sequence of answers. |
| Outcome: | The proposed framework improves performance from 20.8 precision@1 to 27.5 precision@1. |
Cross-functional Analysis of Generalization in Behavioral Learning (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing evaluation paradigms for behavioral learning use correlations in training data, but they ignore important model properties such as fairness. |
| Approach: | They propose an analysis method for evaluating behavioral learning considering generalization across dimensions of different granularity levels. |
| Outcome: | The proposed method optimizes behavior-specific loss functions and evaluates models on several partitions of the behavioral test suite controlled to leave out specific phenomena. |
Exploring Contrast Consistency of Open-Domain Question Answering Systems on Minimally Edited Questions (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing studies on the ability of a model to make consistently correct predictions in the presence of perturbations have not been conducted in open-domain question answering (OpenQA). |
| Approach: | They propose a query-side contrastive loss to improve the dense passage retriever (DPR) to improve DPR training. |
| Outcome: | The proposed approach improves the density of the dense passage retriever (DPR) training set without sacrificing accuracy on standard test sets. |
ParsiNLU: A Suite of Language Understanding Challenges for Persian (2021.tacl-1)
Copied to clipboard
Daniel Khashabi, Arman Cohan, Siamak Shakeri, Pedram Hosseini, Pouya Pezeshkpour, Malihe Alikhani, Moin Aminnaseri, Marzieh Bitaab, Faeze Brahman, Sarik Ghazarian, Mozhdeh Gheini, Arman Kabiri, Rabeeh Karimi Mahabagdi, Omid Memarrast, Ahmadreza Mosallanezhad, Erfan Noury, Shahab Raji, Mohammad Sadegh Rasooli, Sepideh Sadeghi, Erfan Sadeqi Azer, Niloofar Safi Samghabadi, Mahsa Shafaei, Saber Sheybani, Ali Tazarv, Yadollah Yaghoobzadeh
| Challenge: | Despite progress in natural language understanding, most progress is concentrated on resource-rich languages like English . despite high-quality benchmarks, there are few available NLU datasets for Persian language . |
| Approach: | They propose a benchmark for Persian language that includes a range of language understanding tasks . they present their results on monolingual and multilingual pre-trained language models . |
| Outcome: | The proposed benchmarks compare human performance with monolingual and multilingual models on Persian language with high quality evaluation datasets. |
CPRM: A LLM-based Continual Pre-training Framework for Relevance Modeling in Commercial Search (2025.naacl-industry)
Copied to clipboard
Kaixin Wu, Yixin Ji, Zeyuan Chen, Qiang Wang, Cunxiang Wang, Hong Liu, Baijun Ji, Xu Jia, Zhongyi Liu, Jinjie Gu, Yuan Zhou, Linjian Mo
| Challenge: | Relevance modeling between queries and items is a key component of commercial search engines. |
| Approach: | They propose a framework for continual pre-training of LLMs to enhance domain knowledge . they employ queries and multi-field item to jointly pre-train for enhancing domain knowledge. |
| Outcome: | The proposed model achieves convincing performance compared to strong baselines. |
IIRC: A Dataset of Incomplete Information Reading Comprehension Questions (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing reading comprehension tasks focus on questions for which the contexts provide all the information required to answer them, thus not evaluating a system’s performance at identifying a potential lack of sufficient information and locating sources for that information. |
| Approach: | They propose to use a dataset with 13K questions over paragraphs from English Wikipedia that provide only partial information to answer them, with the missing information occurring in one or more linked documents. |
| Outcome: | The proposed model achieves 31.1% F1 on the reading comprehension task, while estimated human performance is 88.4%. |
Identifying Well-formed Natural Language Questions (D18-1)
Copied to clipboard
| Challenge: | Understanding search queries is a hard problem as it involves dealing with “word salad” text. |
| Approach: | They propose to identify a well-formed natural language question from a dataset of 25,100 questions. |
| Outcome: | The proposed classifier improves the performance of neural sequence-to-sequence models for generating questions for reading comprehension. |
Less Is More: Domain Adaptation with Lottery Ticket for Reading Comprehension (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing domain adaptation paradigms for reading comprehension require large amounts of annotation data to achieve the desired task performance. |
| Approach: | They propose a few-shot domain adaptation paradigm for reading comprehension . they introduce self-attention attribution to weigh parameters and refine the lottery subnetwork . |
| Outcome: | The proposed model outperforms the full model fine-tuning adaptation on four out of five domains with a small amount of data available for adaptation. |
Retrieval-guided Counterfactual Generation for QA (2022.acl-long)
Copied to clipboard
| Challenge: | Recent work shows that data augmentation using counterfactuals can help ameliorate this weakness. |
| Approach: | They propose a Retrieve-Generate-Filter technique to generate counterfactuals using QA framework and question generation model trained on original task data. |
| Outcome: | The proposed method improves performance on out-of-domain and challenging evaluation sets over and above existing methods. |
Can We Guide a Multi-Hop Reasoning Language Model to Incrementally Learn at Each Single-Hop? (2022.coling-1)
Copied to clipboard
| Challenge: | Recent developments have shown that pre-trained language models are effective soft reasoners over language. |
| Approach: | They propose to model multi-hop reasoning process as a sequence of explicit single-hop steps. |
| Outcome: | The proposed model improves on multiple-choice question answering and reading comprehension with 68.4% and 16.0% w.r.t. classic PLMs. |
Contrastive Perplexity for Controlled Generation: An Application in Detoxifying Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to generate toxic content by large language models are based on pipelines . current approaches focus on preserving performance while effectively mitigating toxicity . |
| Approach: | They propose a framework for implicit knowledge editing and controlled text generation by using hard negatives. |
| Outcome: | The proposed framework significantly reduces toxic generation while maintaining strong performance on downstream tasks. |
DACIP-RC: Domain Adaptive Continual Instruction Pre-Training via Reading Comprehension on Business Conversations (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been used in real-world industrial scenarios for various natural language processing tasks, but their high inference cost makes their deployment impractical, necessitating the use of smaller models. |
| Approach: | They propose a continual pre-training technique that generates diverse task instructions and responses via reading comprehension on conversation transcripts, enabling better instruction generalization. |
| Outcome: | The proposed technique improves small LLMs’ domain adaptability for business conversational tasks, compared with traditional methods that rely on next-token prediction. |
Benchmarking Machine Reading Comprehension: A Psychological Perspective (2021.eacl-main)
Copied to clipboard
| Challenge: | MRC is a task that tests the ability of a machine to read and understand unstructured text. |
| Approach: | They propose a theoretical basis for the design of MRC datasets based on psychology and psychometrics and propose shortcut-proof questions and explanations as a part of the task design. |
| Outcome: | The proposed datasets should evaluate the model's ability to understand context-dependent situations and ensure substantive validity by shortcut-proof questions and explanation as a part of the task design. |
Seemingly Plausible Distractors in Multi-Hop Reasoning: Are Large Language Models Attentive Readers? (2024.emnlp-main)
Copied to clipboard
| Challenge: | State-of-the-art Large Language Models (LLMs) are accredited with a number of different capabilities, including reading comprehension, mathematical and reasoning skills, and possessing scientific knowledge. |
| Approach: | They propose a benchmark to generate seemingly plausible multi-hop reasoning chains that ultimately lead to incorrect answers. |
| Outcome: | The proposed model circumvents the reasoning requirement but in subtle ways . it shows that it is more difficult to generate plausible alternatives . |
Joint Training of Candidate Extraction and Answer Selection for Reading Comprehension (P18-1)
Copied to clipboard
| Challenge: | Various advanced neural models have been proposed for reading comprehension, but most models ignore its relations with other answer candidates. |
| Approach: | They propose to model reading comprehension as an extract-then-select two-stage procedure . they first extract answer candidates from passages, then select the final answer by combining information from all candidates. |
| Outcome: | The proposed approach improves state-of-the-art performance on open-domain reading comprehension datasets. |
Question Generation for Reading Comprehension Assessment by Modeling How and What to Ask (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing Question Generation systems focus on extractive questions and do not control the type of questions. |
| Approach: | They propose a question generation model that generates inferential questions from text . they propose he model can generate questions annotated with story-based reading comprehension skills . |
| Outcome: | The proposed model outperforms baselines on a reading comprehension dataset. |
Cross-Lingual Machine Reading Comprehension (D19-1)
Copied to clipboard
| Challenge: | Existing work on machine reading comprehension task is focused on English, but there are few efforts on other languages due to the lack of large-scale training data. |
| Approach: | They propose a cross-lingual machine reading comprehension task for other languages . they propose cloze-style reading comprehension and various neural network approaches . |
| Outcome: | The proposed model improves reading comprehension performance of Chinese datasets over state-of-the-art systems by a large margin over existing systems. |
A Multi-Type Multi-Span Network for Reading Comprehension that Requires Discrete Reasoning (D19-1)
Copied to clipboard
| Challenge: | Existing models for reading comprehension and question answering do not support discrete reasoning abilities. |
| Approach: | They propose a reading comprehension model that uses a multi-type answer predictor and a multiple-span extraction method to produce one or multiple text spans. |
| Outcome: | The proposed model achieves 79.9 F1 on the DROP hidden test set, creating new state-of-the-art results. |
Challenging Reading Comprehension on Daily Conversation: Passage Completion on Multiparty Dialog (N18-1)
Copied to clipboard
| Challenge: | Existing approaches to reading comprehension on multiparty dialogs have focused on children's stories or newswire. |
| Approach: | They propose a new corpus and a robust deep learning architecture for a task in reading comprehension on multiparty dialog. |
| Outcome: | The proposed model outperforms the state-of-the-art model on a different genre using bidirectional LSTM, showing a 13.0+% improvement for longer dialogs. |
Context-Aware Answer Extraction in Question Answering (2020.emnlp-main)
Copied to clipboard
| Challenge: | Extractive QA models have shown promising performance in predicting the correct answer to a given question. |
| Approach: | They propose a BLANC-based context prediction task that learns the context prediction tasks. |
| Outcome: | The proposed model outperforms the state-of-the-art models on reading comprehension and hotpotQA. |
Evaluating Large Language Models on Controlled Generation Tasks (2023.emnlp-main)
Copied to clipboard
Jiao Sun, Yufei Tian, Wangchunshu Zhou, Nan Xu, Qian Hu, Rahul Gupta, John Wieting, Nanyun Peng, Xuezhe Ma
| Challenge: | Recent studies have looked into the ability of large language models in various benchmark tasks, including question generation, reading comprehension, multilingual and etc. However, few studies investigate the controllability of large languages. |
| Approach: | They propose to compare large language models with state-of-the-start finetuned smaller models to find that large language model controls are comparable to smaller models. |
| Outcome: | The proposed model can meet hard constraints and perform better than state-of-the-art models. |
What do Models Learn from Question Answering Datasets? (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models have outperformed humans on question answering datasets, but they have yet to outperform humans on the task of question answering itself. |
| Approach: | They evaluate BERT-based question answering models on their generalizability to out-of-domain examples, responses to missing or incorrect data, and ability to handle question variations. |
| Outcome: | The proposed models outperform human baselines on the widely-used SQuAD 1.1 and SQu AD 2.0 datasets. |
WYWEB: A NLP Evaluation Benchmark For Classical Chinese (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for classical Chinese are inadequate to evaluate performance of different NLP models. |
| Approach: | They propose an evaluation benchmark for classical Chinese NLP, which evaluates existing models. |
| Outcome: | The proposed benchmark evaluates the performance of existing models in classical Chinese. |
Measuring and Improving Attentiveness to Partial Inputs with Counterfactuals (2024.findings-emnlp)
Copied to clipboard
Yanai Elazar, Bhargavi Paranjape, Hao Peng, Sarah Wiegreffe, Khyathi Chandu, Vivek Srikumar, Sameer Singh, Noah Smith
| Challenge: | Existing studies have found that datasets with paired inputs are prone to spurious correlations, resulting in models trained only on those outperform chance. |
| Approach: | They propose a counterfactual attentiveness test to measure reliance on spurious correlations by replacing part of the input with its counterpart from a different example. |
| Outcome: | The proposed method improves models' attentiveness on ten datasets spanning four tasks: natural language inference, reading comprehension, paraphrase detection, and visual & language reasoning. |
Retrieve, Read, Rerank: Towards End-to-End Multi-Document Reading Comprehension (P19-1)
Copied to clipboard
| Challenge: | Existing approaches to answer reading comprehension tasks are inefficient since the input is re-encoded within each module. |
| Approach: | They propose a unified question answering model that combines context retrieving, reading comprehension, and answer reranking to predict the final answer. |
| Outcome: | The proposed model outperforms the baseline model and achieves state-of-the-art results on two versions of TriviaQA and two variants of SQuAD. |
XQA: A Cross-lingual Open-domain Question Answering Dataset (P19-1)
Copied to clipboard
| Challenge: | Open-domain question answering aims to answer questions through text retrieval and reading comprehension . but, the success of these models relies on a massive volume of training data, which is not available in other languages . a new dataset aims at investigating cross-lingual OpenQA . |
| Approach: | They propose to use a dataset for cross-lingual OpenQA research to test models . they use XQA dataset to train models with large volumes of labeled data . |
| Outcome: | The proposed model achieves best results in almost all target languages while the performance is lower than that of English. |
Bi-directional CognitiveThinking Network for Machine Reading Comprehension (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for reading comprehension are still in their infancy at the level of cognitive intelligence. |
| Approach: | They propose a bi-directional cognitive knowledge framework to simulate reverse thinking and inertial thinking in the brain to answer questions. |
| Outcome: | The proposed framework shows that bi-directional knowledge helps the QA task. |
Question Answering by Reasoning Across Documents with Graph Convolutional Networks (N19-1)
Copied to clipboard
| Challenge: | Recent research in reading comprehension has focused on answering questions based on individual documents or even single paragraphs. |
| Approach: | They propose a neural model which integrates and reasons relying on information spread within documents and across multiple documents. |
| Outcome: | The proposed model achieves state-of-the-art on a multi-document question answering dataset, WikiHop. |
ForceReader: a BERT-based Interactive Machine Reading Comprehension Model with Attention Separation (2020.coling-main)
Copied to clipboard
| Challenge: | Various BERT-based reading comprehension models have been proposed, however, these models employ the combined input method without further modification for reading comprehension. |
| Approach: | They propose a BERT-based interactive machine reading comprehension model that uses BERT's combined input method without further modification for reading comprehension. |
| Outcome: | The proposed model improves reading comprehension tasks compared to BERT-based models. |
Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning (D19-1)
Copied to clipboard
| Challenge: | Existing reading comprehension datasets focus on factual and literal understanding of context paragraphs, but our dataset focuses on reading between the lines over a diverse collection of everyday narratives. |
| Approach: | They propose a large-scale dataset that requires commonsense-based reading comprehension, formulated as multiple-choice questions. |
| Outcome: | The proposed architecture improves over the baselines of existing reading comprehension datasets and shows a significant gap between machine (68.4%) and human performance (94%). |
Span Selection Pre-training for Question Answering (2020.acl-main)
Copied to clipboard
Michael Glass, Alfio Gliozzo, Rishav Chakravarti, Anthony Ferritto, Lin Pan, G P Shrivatsa Bhargav, Dinesh Garg, Avi Sil
| Challenge: | Pre-trained BERTs provide large gains across many language understanding tasks, achieving a new state-of-the-art (SOTA). |
| Approach: | They propose a new pre-training task inspired by reading comprehension to better align the pre- training from memorization to understanding. |
| Outcome: | The proposed model outperforms BERT-BASE and BERT LARGE on a new dataset and improves answer prediction F1 by 4 points and supporting fact prediction F1. |
BiPaR: A Bilingual Parallel Dataset for Multilingual and Cross-lingual Reading Comprehension on Novels (D19-1)
Copied to clipboard
| Challenge: | Using BiPaR, we build monolingual, multilingual and cross-lingual MRC on novels. |
| Approach: | They propose a bilingual parallel novel-style machine reading comprehension dataset BiPaR . they collect 3,667 bilingual parallel paragraphs from Chinese and English novels . |
| Outcome: | The proposed dataset supports multilingual and cross-lingual reading comprehension. |
Discourse-Aware Semantic Self-Attention for Narrative Reading Comprehension (D19-1)
Copied to clipboard
| Challenge: | Existing work on discourse-aware self-attention models for reading comprehension uses annotations . |
| Approach: | They propose to use linguistic annotations as a basis for a Discourse-Aware Semantic Self-Attention encoder for reading comprehension on narrative texts. |
| Outcome: | The proposed model improves reading comprehension performance on narrative texts up to +3.4 Rouge-L . it also improves inter- and cross-sentential discourse relations, sentence-internal semantic role relations, and long-distance coreference relations. |
Logic-guided Semantic Representation Learning for Zero-Shot Relation Classification (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods to zero-shot relation classification can only identify seen relations . existing methods rely on descriptive information to improve understandability of relation types . |
| Approach: | They propose a logic-guided semantic representation learning model for zero-shot relation classification that builds connections between seen and unseen relations via implicit and explicit semantic representations with knowledge graph embeddings and logic rules. |
| Outcome: | The proposed model can generalize to unseen relation types and achieve promising improvements. |
Domain Adapted Distant Supervision for Pedagogically Motivated Relation Extraction (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing systems for question generation based on shallow linguistic information at text level generate questions for reading comprehension. |
| Approach: | They propose a relation extraction system that extracts pedagogically motivated relation types from a text to generate questions for reading comprehension. |
| Outcome: | The proposed system extracts pedagogically motivated relations from a text . it performs better on the subset of relations suitable for pedagical purposes. |
Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models (2024.acl-long)
Copied to clipboard
Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, Kaidi Xu
| Challenge: | Large Language Models (LLMs) show promising results in language generation but often “hallucinate”, making their outputs less reliable. |
| Approach: | They propose to shift attention to more relevant components at token- and sentence-levels for better UQ. |
| Outcome: | The proposed approach improves the performance of a range of popular “off-the-shelf” LLMs with model sizes extending up to 33B parameters. |
What’s Missing: A Knowledge Gap Guided Approach for Multi-hop Question Answering (D19-1)
Copied to clipboard
| Challenge: | Multi-hop textual question answering requires combining information from multiple sentences. |
| Approach: | They propose a model that explicitly identifies the knowledge gap between a key span in the provided knowledge and the answer choices. |
| Outcome: | The proposed model outperforms existing models on the OpenBookQA dataset. |
TabComp: A Dataset for Visual Table Reading Comprehension (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing OCR-free models struggle with complex table layouts and formatting. |
| Approach: | They propose a dataset to evaluate OCR-free models in table reading comprehension using table images, questions, and generative answers. |
| Outcome: | The proposed model outperforms existing models in the visual table reading comprehension dataset and encourages the research community to work on improving them. |
Lexical Generalization Improves with Larger Models and Longer Training (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Excessive utilization of lexical overlap heuristics can lead to failure on challenging inputs. |
| Approach: | They analyze the use of lexical overlap heuristics in natural language inference, paraphrase detection, and reading comprehension using a contrastive dataset. |
| Outcome: | The proposed model can be used to improve performance on a wide range of tasks, but it is often wrong. |
Benchmarking Large Language Models on CFLUE - A Chinese Financial Language Understanding Evaluation Dataset (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models have revolutionized natural language processing (NLP) there is an urgent need for new benchmarks to keep pace with the development of LLMs. |
| Approach: | They propose a benchmark to assess the capability of large language models (LLMs) they use a dataset to provide both knowledge assessment and application assessment . |
| Outcome: | The proposed benchmark provides datasets tailored for knowledge assessment and application assessment. |
Abstract Meaning Representation-Based Logic-Driven Data Augmentation for Logical Reasoning (2024.findings-acl)
Copied to clipboard
Qiming Bao, Alex Peng, Zhenyun Deng, Wanjun Zhong, Gael Gendron, Timothy Pistotti, Neset Tan, Nathan Young, Yang Chen, Yonghua Zhu, Paul Denny, Michael Witbrock, Jiamou Liu
| Challenge: | Empirical evidence shows that our proposed method improves performance across seven downstream tasks. |
| Approach: | They propose a logic-driven data augmentation approach that converts text into AMR graphs and converts them back into text to create augmented data. |
| Outcome: | The proposed method leads on the ReClor leaderboard and improves on seven downstream tasks. |
FCM: A Fine-grained Comparison Model for Multi-turn Dialogue Reasoning (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing neural dialogue models only capture syntactic and semantic information, but fail to model the logical consistency between the dialogue history and the generated response. |
| Approach: | They propose a fine-grained comparison model to capture syntactic and semantic information and then compare each candidate's representation with the whole history to obtain a history consistency representation. |
| Outcome: | The proposed model obtains higher ranking scores than baseline models on two public dialogue datasets. |
Learning Event Graph Knowledge for Abductive Reasoning (2021.acl-long)
Copied to clipboard
| Challenge: | Existing models for abductive reasoning based on formal logic lack commonsense knowledge and effective reasoning mechanism. |
| Approach: | They propose a narrative text-based abductive reasoning task NLI with a latent variable to capture commonsense knowledge from event graph for guiding the abductive reasoning task. |
| Outcome: | The proposed model outperforms baseline methods on the abductive reasoning task. |
HICD: Hallucination-Inducing via Attention Dispersion for Contrastive Decoding to Mitigate Hallucinations in Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) often generate hallucinations, producing outputs that are contextually inaccurate or factually incorrect. |
| Approach: | They propose a method that selects attention heads crucial to the model's prediction as inducing heads and induces hallucinations by dispersing attention of these inducers. |
| Outcome: | The proposed method significantly improves performance on tasks requiring contextual faithfulness, reading comprehension, and question answering. |
Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment (2024.acl-long)
Copied to clipboard
| Challenge: | Existing work cheaply emulates LLMs, allowing users to create profiles for their preferred characters. |
| Approach: | They propose a self-alignment method that encourages an instruction-following LLM to simulate role-play dialogues as a variant of reading comprehension. |
| Outcome: | The proposed model outperforms open-source role-play benchmarks and the roleplay subset of MT-Bench in multiple parameters. |
Episodic Memory Reader: Learning What to Remember for Question Answering from Streaming Data (P19-1)
Copied to clipboard
| Challenge: | Existing QA methods lack scalability and performance is difficult to solve with document-level contexts. |
| Approach: | They propose an end-to-end deep network model that sequentially reads the input contexts into an external memory while replacing memories that are less important for answering unseen questions. |
| Outcome: | The proposed model improves on a synthetic dataset and real-world large-scale textual and video QA datasets. |
Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Recent models for unsupervised representation learning of text have put little focus on discourse-level representations. |
| Approach: | They propose an inter-sentence objective for pretraining language models that models discourse coherence and the distance between sentences. |
| Outcome: | The proposed model outperforms the BERT-Large model on the discourse representation benchmark DiscoEval and yields gains of 2%-6% absolute even for tasks that do not explicitly evaluate discourse. |
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization (2020.acl-main)
Copied to clipboard
| Challenge: | Existing automatic metrics do not capture errors in abstractive summarization models. |
| Approach: | They propose an automatic question answering metric for faithfulness that leverages recent advances in reading comprehension. |
| Outcome: | The proposed metric has significantly higher correlation with human faithfulness scores on highly abstracted summaries. |
A Nil-Aware Answer Extraction Framework for Question Answering (D18-1)
Copied to clipboard
| Challenge: | Recent research suggests that reading comprehension-based question answering systems assume that every question has a valid answer in the associated passage. |
| Approach: | They propose a novel nil-aware answer span extraction framework that can return Nil or a text span from the associated passage as an answer in a single step. |
| Outcome: | The proposed framework outperforms baseline approaches on a newsQA dataset. |
MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension (P19-1)
Copied to clipboard
| Challenge: | A large number of reading comprehension (RC) datasets have been created, but little research has been done on whether they generalize to one another and the extent to which existing datasets can be leveraged for improving performance on new ones. |
| Approach: | They propose a BERT-based reading comprehension model that can be trained on multiple RC datasets. |
| Outcome: | The proposed model can be trained on multiple RC datasets and improve performance on five RC data. |
Simple and Effective Curriculum Pointer-Generator Networks for Reading Comprehension over Long Narratives (P19-1)
Copied to clipboard
Yi Tay, Shuohang Wang, Anh Tuan Luu, Jie Fu, Minh C. Phan, Xingdi Yuan, Jinfeng Rao, Siu Cheung Hui, Aston Zhang
| Challenge: | Using a pointer-generator framework for reading/sampling over large documents, we propose a framework for learning over long narratives where documents easily span over thousands of tokens. |
| Approach: | They propose a curriculum learning (CL) based pointer-generator framework for reading/sampling over large documents, enabling diverse training of the neural model based on the notion of alternating contextual difficulty. |
| Outcome: | The proposed framework improves on the NarrativeQA reading comprehension benchmark and reaches state-of-the-art performance. |
It Is Not About What You Say, It Is About How You Say It: A Surprisingly Simple Approach for Improving Reading Comprehension (2024.findings-acl)
Copied to clipboard
| Challenge: | Experimenting with 9 large language models across 3 datasets, emphasizing the context yields superior results compared to question emphasis. |
| Approach: | They ask: How does the order of inputs affect model performance? |
| Outcome: | Experiments with 9 large language models show that emphasizing the question and context improves model performance. |
STARC: Structured Annotations for Reading Comprehension (2020.acl-main)
Copied to clipboard
| Challenge: | STARC is an annotation framework for reading comprehension with multiple choice questions . existing annotation frameworks have limited support for reading behavior analyses . |
| Approach: | They propose a new annotation framework for assessing reading comprehension with multiple choice questions . they use a span ablation dataset to demonstrate that it can be leveraged for a key new application . |
| Outcome: | The proposed framework can be leveraged for a key new application for SAT-like reading comprehension materials. |
Improving Dialogue Discourse Parsing via Reply-to Structures of Addressee Recognition (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to learn dialogue discourse parsing with related tasks require additional annotation, thus limiting their generality. |
| Approach: | They propose a multitasking framework that integrates dialogue discourse parsing with addressee recognition to reflect relation-based structure of dialogue. |
| Outcome: | The proposed framework outperforms baselines on the Molweni and STAC datasets. |
MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metrics (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing reading comprehension metrics rely on token overlap and are agnostic to the nuances of reading comprehension. |
| Approach: | They propose a benchmark for training and evaluating generative reading comprehension metrics: MOdeling Correctness with Human Annotations. |
| Outcome: | The proposed benchmark outperforms baseline metrics by 10 to 36 absolute Pearson points on held-out annotations. |
How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks (D18-1)
Copied to clipboard
| Challenge: | Recent research addresses reading comprehension, where examples consist of (question, passage, answer) tuples. |
| Approach: | They establish sensible baselines for bAbI, SQuAD, CBT, CNN and Who-did-What datasets and compare them to their previous work. |
| Outcome: | The proposed models perform on 14 out of 20 bAbI, SQuAD, CBT, CNN and Who-did-What datasets. |
VMLU Benchmarks: A comprehensive benchmark toolkit for Vietnamese LLMs (2025.acl-long)
Copied to clipboard
Cuc Thi Bui, Nguyen Truong Son, Truong Van Trang, Lam Viet Phung, Pham Nhut Huy, Hoang Anh Le, Quoc Huu Van, Phong Nguyen-Thuan Do, Van Le Tran Truc, Duc Thanh Chau, Le-Minh Nguyen
| Challenge: | The evolution of Large Language Models (LLMs) has underscored the need for benchmarks designed for various languages and cultural contexts. |
| Approach: | They propose to use Vietnamese multitask language understanding (VMLU) benchmarks to assess different capabilities of LLMs, including general knowledge, reading comprehension, reasoning, and conversational skills. |
| Outcome: | The VMLU Benchmarks assess LLMs' general knowledge, reading comprehension, reasoning, and conversational skills. |
2M-BELEBELE: Highly Multilingual Speech and American Sign Language Comprehension Dataset Download PDF (2025.findings-acl)
Copied to clipboard
Marta R. Costa-jussà, Bokai Yu, Pierre Andrews, Belen Alastruey, Necati Cihan Camgoz, Joe Chuang, Jean Maillard, Christophe Ropers, Arina Turkatenko, Carleigh Wood
| Challenge: | We extend the BELEBELE dataset to speech and sign, and extend the Automatic Speech Recognition Benchmark, FLEURS, by 20%. |
| Approach: | They extend the BELEBELE and FLEURS speech comprehension datasets to speech and sign . they evaluate the datasets for 5-shot and zero-shot settings and find that the accuracy is 10% lower than reading comprehension. |
| Outcome: | The proposed dataset covers 91 spoken languages and one sign language (ASL) it also extends the Automatic Speech Recognition Benchmark, FLEURS, by 20% across languages. |
Stimulate the Critical Thinking of LLMs via Debiasing Discussion (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies show that large language models (LLMs) are often prone to stance homogeneity and human preference biases when faced with conflicting perspectives. |
| Approach: | They propose a novel two-stage training framework to address stance homogeneity bias and human preference bias by generating multi-model discussion datasets and optimizing reinforcement learning from human feedback to align with discussion correctness. |
| Outcome: | The proposed framework reduces stance homogeneity bias and human preference bias and improves generalization capabilities on non-discussion scenarios and out-of-domain datasets. |
Automated Focused Feedback Generation for Scientific Writing Assistance (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work has focused on improving surface form and style rather than manuscript content. |
| Approach: | They propose to use a scientific writing focused feedback tool to generate specific, actionable and coherent comments which identify weaknesses in a paper and/or propose revisions to it. |
| Outcome: | The proposed tool outperforms existing approaches in specificity, reading comprehension and overall helpfulness of the generated reviews. |
Evaluating and Enhancing the Robustness of Neural Network-based Dependency Parsing Models with Adversarial Examples (2020.acl-main)
Copied to clipboard
| Challenge: | Previously studies focused on semantic tasks such as sentiment analysis, question answering and reading comprehension. |
| Approach: | They propose two approaches to study where and how adversarial examples exist in dependency parsing . they use a state-of-the-art parser to find adversarials in existing texts . |
| Outcome: | The proposed approaches show that adversarial examples exist in dependency parsing . they show that up to 77% of input examples admit adversarials . |
ESTER: A Machine Reading Comprehension Dataset for Reasoning about Event Semantic Relations (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent event-centric reading comprehension datasets focus mostly on event arguments or temporal relations. |
| Approach: | They propose a machine reading comprehension dataset that leverages natural language queries to reason about the five most common event semantic relations. |
| Outcome: | The proposed dataset shows that current SOTA systems achieve 22.1%, 63.3% and 83.5% for token-based exact-match, **F1** and event-based **HIT@1** scores. |
GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level (D19-1)
Copied to clipboard
| Challenge: | SQA is an emerging application of NLP in the medical, geography, and legal domains. |
| Approach: | They propose a dataset of 1,981 scenarios and 4,110 multiple-choice questions in geography domain at high school level. |
| Outcome: | The proposed dataset consists of 1,981 scenarios and 4,110 multiple-choice questions in the geography domain at high school level. |
How to Engage your Readers? Generating Guiding Questions to Promote Active Reading (2024.acl-long)
Copied to clipboard
| Challenge: | Using questions in written text is an effective strategy to enhance readability, but what makes an active reading question good, what the linguistic role of these questions is, and what is their impact on human reading remains understudied. |
| Approach: | They present a dataset of 10K in-text questions from textbooks and scientific articles and explore various approaches to generate such questions using language models. |
| Outcome: | The generated questions are of high quality and are almost as effective as human-written questions in terms of improving readers’ memorization and comprehension. |
Multi-source Meta Transfer for Low Resource Multiple-Choice Question Answering (2020.acl-main)
Copied to clipboard
| Challenge: | Existing MCQA datasets are small in size, which increases difficulty of model learning and generalization. |
| Approach: | They propose a multi-source meta transfer framework for low-resource multiple-choice question answering . they extend meta learning by incorporating multiple training sources to learn a generalized feature representation across domains . |
| Outcome: | The proposed framework is independent of backbone language models and can bridge the distribution gap between training sources and target. |
FastMem: Fast Memorization of Prompt Improves Context Awareness of Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models struggle with context awareness, leading to inaccuracies in tasks requiring faithful adherence to provided information. |
| Approach: | They propose a method to enhance LLMs' context awareness by updating only the last Feed-Forward Network module to maximize the likelihood of the prompt before inference . |
| Outcome: | The proposed method improves the accuracy of Llama 3-8B-Inst on the NQ-SWAP dataset from 59.1% to 71.6% and reduces the output structure failure rate of Qwen 1.5-4B-Chat from 34.9% to 25.5%. |
What’s the Meaning of Superhuman Performance in Today’s NLU? (2023.acl-long)
Copied to clipboard
Simone Tedeschi, Johan Bos, Thierry Declerck, Jan Hajič, Daniel Hershcovich, Eduard Hovy, Alexander Koller, Simon Krek, Steven Schockaert, Rico Sennrich, Ekaterina Shutova, Roberto Navigli
| Challenge: | Recent research has focused on developing larger pretrained language models and introducing benchmarks such as SuperGLUE and SQuAD to measure their abilities. |
| Approach: | They propose to use benchmarks such as SuperGLUE and SQUAD to evaluate PLMs' abilities in language understanding, reasoning, and reading comprehension to assess their performance. |
| Outcome: | The proposed benchmarks have serious limitations affecting comparison between humans and PLMs and provide recommendations for fairer and more transparent benchmarks. |
DaMoC: Efficiently Selecting the Optimal Large Language Model for Fine-tuning Domain Tasks Based on Data and Model Compression (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models excel in general tasks but struggle with domain-specific ones, requiring fine-tuning with specific data. |
| Approach: | They propose a Data and Model Compression Framework that categorizes data filtering methodologies into three distinct paradigms: (1) distribution-aware methods, (2) quality-a aware methods, and (3) hybrid approaches considering both dimensions. |
| Outcome: | The proposed framework can select the optimal LLM while saving approximately 20-fold in training time. |
HAE-RAE Bench: Evaluation of Korean Knowledge in Language Models (2024.lrec-main)
Copied to clipboard
Guijin Son, Hanwool Lee, Suwan Kim, Huiseo Kim, Jae cheol Lee, Je Won Yeom, Jihyu Jung, Jung woo Kim, Songseong Kim
| Challenge: | Existing evaluation tools rely on translations of English datasets or translation-specific benchmarks such as WMT 21 to assess large language models. |
| Approach: | They propose a dataset curated to challenge models lacking Korean cultural and contextual depth. |
| Outcome: | The HAE-RAE Bench challenges models lacking Korean cultural and contextual depth by highlighting their aptitude for recalling Korean-specific knowledge and cultural contexts. |
Null-Shot Prompting: Rethinking Prompting Large Language Models With Hallucination (2024.emnlp-main)
Copied to clipboard
| Challenge: | a series of investigations into an interesting phenomenon where performance increases in large language models when providing a prompt that causes and exploits hallucination. |
| Approach: | They propose a null-shot prompting approach that intentionally instructs LLMs to look at and utilize information from a nil section. |
| Outcome: | The proposed approach causes and exploits hallucination in large language models on a range of tasks including arithmetic reasoning, commonsense reasoning, and reading comprehension. |
InternalInspector I2: Robust Confidence Estimation in LLMs through Internal States (2024.findings-emnlp)
Copied to clipboard
Mohammad Beigi, Ying Shen, Runing Yang, Zihao Lin, Qifan Wang, Ankith Mohan, Jianfeng He, Ming Jin, Chang-Tien Lu, Lifu Huang
| Challenge: | Large Language Models (LLMs) often struggle with generating reliable outputs, often producing high-confidence inaccuracies known as hallucinations. |
| Approach: | They propose a framework that leverages contrastive learning on internal states including attention states, feed-forward states, and activation states of all layers to enhance confidence estimation in LLMs. |
| Outcome: | The framework outperforms existing methods in the hallucination detection benchmark HaluEval and the previous methods at the same time. |
Numerical reasoning in machine reading comprehension tasks: are we there yet? (2021.emnlp-main)
Copied to clipboard
| Challenge: | Numerical reasoning based machine reading comprehension models have achieved near-human performance on a variety of benchmarks, but are they capable of learning to reason? |
| Approach: | They propose to use a DROP benchmark to measure machine reading comprehension and investigate models that have achieved near-human performance over standard metrics. |
| Outcome: | The DROP benchmark has inspired the design of specialized BERT and embedding the results into a specialized model. |
Shaking Up VLMs: Comparing Transformers and Structured State Space Models for Vision & Language Modeling (2024.emnlp-main)
Copied to clipboard
| Challenge: | a task-agnostic visual encoding yields minimal performance gains on grounding, but Transformers outperform Mamba at in-context multimodal retrieval. |
| Approach: | They propose to replace Transformers in Visual Language Models with Mamba, a structured state space model that demonstrates promising performance in sequence modeling. |
| Outcome: | The proposed model outperforms Transformers-based models in captioning, question answering, and reading comprehension. |
KazQAD: Kazakh Open-Domain Question Answering Dataset (2024.lrec-main)
Copied to clipboard
| Challenge: | KazQAD contains just under 6,000 unique questions with extracted short answers and nearly 12,000 passage-level relevance judgements. |
| Approach: | They introduce a Kazakh open-domain question answering dataset that can be used in reading comprehension and full ODQA settings. |
| Outcome: | The proposed dataset can be used in reading comprehension and full ODQA settings, as well as for information retrieval experiments. |
Developing Dataset of Japanese Slot Filling Quizzes Designed for Evaluation of Machine Reading Comprehension (2020.lrec-1)
Copied to clipboard
| Challenge: | a novel dataset of slot filling quizzes is developed for evaluation of machine reading comprehension . the dataset consists of quizzed characters of target books as slots . |
| Approach: | They develop a Japanese dataset of slot filling quizzes for machine reading comprehension . they define each quiz as a 4-tuple: context passage, query holding a slot, answer character and set of possible answer characters . |
| Outcome: | The proposed dataset includes a 4-tuple quiz, a context passage, q and a query . the q&a is performed on characters of target books, which play important roles in narrative texts . |
LORAXBENCH: A Multitask, Multilingual Benchmark Suite for 20 Indonesian Languages (2025.emnlp-main)
Copied to clipboard
| Challenge: | LORAXBENCH is a benchmark for low-resource languages of Indonesia . it covers reading comprehension, open domain QA, language inference, causal reasoning, translation, and cultural question answering across 20 languages. |
| Approach: | They propose a benchmark that focuses on low-resource languages of Indonesia and covers 6 diverse tasks: reading comprehension, open-domain QA, language inference, causal reasoning, translation, and cultural question answering. |
| Outcome: | The proposed benchmark covers reading comprehension, open-domain QA, language inference, causal reasoning, translation, and cultural question answering across 20 Indonesian languages. |
Multi-Level Information Retrieval Augmented Generation for Knowledge-based Visual Question Answering (2024.emnlp-main)
Copied to clipboard
| Challenge: | Knowledge-Aware Visual Question Answering about Entity tasks require two separate steps to generate accurate answers. |
| Approach: | They propose a multi-level information RAG approach that enhances answer generation through entity retrieval and query expansion. |
| Outcome: | The proposed approach improves answer generation through entity retrieval and query expansion. |
Mitigating Catastrophic Forgetting in Large Language Models with Forgetting-aware Pruning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have shown impressive capabilities in various downstream tasks but typically face Catastrophic Forgetting (CF) during fine-tuning. |
| Approach: | They propose a pruning-based approach to balance CF and downstream task performance by integrating the ratio of the task vector to pre-trained model parameters into the pruning criteria. |
| Outcome: | The proposed pruning-based approach limits CF to just 0.25% while maintaining 99.67% accuracy on downstream tasks. |
From Understanding to Generation: An Efficient Shortcut for Evaluating Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Iterative evaluation of large language models during training can be time- and compute-intensive. |
| Approach: | They reformulate generative tasks into computationally cheaper NLU alternatives and test their performance correlation between them. |
| Outcome: | The proposed alternatives reduce evaluation time by 35x compared to NLU benchmarks. |
SciDQA: A Deep Reading Comprehension Dataset over Scientific Papers (2024.emnlp-main)
Copied to clipboard
| Challenge: | SciDQA is a dataset for question-answering that challenges language models to deeply understand scientific articles. |
| Approach: | They propose a new dataset for reading comprehension that challenges language models to deeply understand scientific articles consisting of 2,937 QA pairs. |
| Outcome: | The SciDQA dataset is based on 2,937 QA pairs and decontextualizes the content, tracks the source document across different versions, and incorporates a bibliography for multi-document question-answering. |
S3Prompt: Instructing the Model with Self-calibration, Self-recall and Self-aggregation to Improve In-context Learning (2024.lrec-main)
Copied to clipboard
| Challenge: | Large language models have limitations in practical applications, such as unsupervised generation and recall of in-context examples. |
| Approach: | They propose a self-calibration, self-recall and self-aggregation prompt pipeline to solve these problems. |
| Outcome: | The proposed pipeline improves the performance of large language models without annotating datasets and model parameter updates. |
Predicting Prosodic Boundaries for Children’s Texts (2025.emnlp-main)
Copied to clipboard
| Challenge: | Using a dataset of 54 leveled English stories annotated for potential pauses, we find that nearly 30% of pause occur at non-punctuation locations of the text. |
| Approach: | They propose to use a text-based model to predict pause locations in children's reading material using a curated dataset of 54 leveled English stories annotated for potential pauses, or prosodic boundaries, by 21 fluent speakers. |
| Outcome: | The proposed model can model both allowed and “forbidden” pauses . it uses a curated dataset of 54 leveled English stories annotated for potential pause locations by 21 fluent speakers . |